Methods and apparatus for using range data to predict object features

ABSTRACT

Embodiments relate to predicting height information for an object. First distance data is determined at a first time when an object is at a first position that is only partially within the field-of-view. Second distance data is determined at a second, later time when the object is at a second, different position that is only partially within the field-of-view. A distance measurement model that models a physical parameter of the object within the field-of-view is determined for the object based on the first and second distance data. Third distance data indicative of an estimated distance to the object prior to the object being entirely within the field-of-view of the distance sensing device is determined based on the first distance data, the second distance data, and the distance measurement model. Data indicative of a height of the object is determined based on the third distance data.

TECHNICAL FIELD

The techniques described herein relate generally to methods andapparatus for using range data to predict object features, including topredict the height of the object.

BACKGROUND

Vision systems can be used in a wide range of applications to performtasks such as measuring objects, inspecting objects, aligning objects,and/or decoding symbology (e.g., barcodes). Such systems often includean image sensor that acquires images of the subject or object, and oneor more processors (e.g., on-board and/or interconnected to the imagesensor) that processes the acquired images. The image information can beprovided as an array of image pixels that each have various colorsand/or intensities. The vision system can be configured to generate adesired output based upon the processed images. For example, for abarcode reader, the imaging device can image an object that may containone or more barcodes. The system can then process the image to identifybarcode, which the system can then decode using a barcode decodingprocess.

SUMMARY

The techniques described herein, in some embodiments, relate to using adistance sensing device, such as a time-of-flight sensor, to measurerange data to predict features of a moving object prior to when suchfeatures can be determined using existing techniques. In someembodiments, the techniques use the distance data to predict the heightof the moving object to focus a lens assembly so that an imaging device(viewing a scene through the lens) can image the moving object at asufficient focus that allows features of the object to be processedbased on the images. For example, while a time-of-flight sensor may onlybe able to accurately measure the distance to an object when the objectis entirely within the field of view of the time-of-flight sensor, thetechniques can provide for predicting the distance to the object wellbefore the object being entirely within the field of view of thetime-of-flight sensor. The techniques can allow a machine vision systemto process features such as physical characteristics of the object(e.g., object size, object area, spacing among objects, etc.), barcodeson the object, perform object inspection, and/or the like.

Some embodiments relate to a computerized method. The method includesaccessing first distance data determined by a distance sensing device ata first time, wherein the distance sensing device determined the firstdistance data with an object at a first position within a field-of-viewof the distance sensing device, wherein the object is only partially inthe field-of-view at the first position. The method includes accessingsecond distance data determined by the distance sensing device at asecond time occurring after the first time, wherein the distance sensingdevice determined the second distance data with the object at a secondposition within the field-of-view of the distance sensing device,wherein the first position is different than the second position, andthe object is only partially in the field-of-view at the secondposition. The method includes determining a distance measurement modelfor the object based on the first distance data and the second distancedata, wherein the distance measurement model is configured to model aphysical parameter of the object within the field-of-view of thedistance sensing device over time. The method includes determining,based on the first distance data, the second distance data, and thedistance measurement model, third distance data indicative of anestimated distance to the object prior to the object being entirelywithin the field-of-view of the distance sensing device. The methodincludes determining, based on the third distance data, data indicativeof a height of the object.

Some embodiments relate to an apparatus. The apparatus includes aprocessor in communication with a memory. The processor is configured toexecute instructions stored in the memory that cause the processor toaccess first distance data determined by a distance sensing device at afirst time, wherein the distance sensing device determined the firstdistance data with an object at a first position within a field-of-viewof the distance sensing device, wherein the object is only partially inthe field-of-view at the first position. The processor is configured toexecute instructions stored in the memory that cause the processor toaccess second distance data determined by the distance sensing device ata second time occurring after the first time, wherein the distancesensing device determined the second distance data with the object at asecond position within the field-of-view of the distance sensing device,wherein the first position is different than the second position, andthe object is only partially in the field-of-view at the secondposition. The processor is configured to execute instructions stored inthe memory that cause the processor to determine a distance measurementmodel for the object based on the first distance data and the seconddistance data, wherein the distance measurement model is configured tomodel a physical parameter of the object within the field-of-view of thedistance sensing device over time. The processor is configured toexecute instructions stored in the memory that cause the processor todetermine, based on the first distance data, the second distance data,and the distance measurement model, third distance data indicative of anestimated distance to the object prior to the object being entirelywithin the field-of-view of the distance sensing device. The processoris configured to execute instructions stored in the memory that causethe processor to determine, based on the third distance data, dataindicative of a height of the object.

Some embodiments relate to at least one non-transitory computer-readablestorage medium. The non-transitory computer-readable storage mediumstores processor-executable instructions that, when executed by at leastone computer hardware processor, cause the at least one computerhardware processor to access first distance data determined by adistance sensing device at a first time, wherein the distance sensingdevice determined the first distance data with an object at a firstposition within a field-of-view of the distance sensing device, whereinthe object is only partially in the field-of-view at the first position.The non-transitory computer-readable storage medium storesprocessor-executable instructions that, when executed by at least onecomputer hardware processor, cause the at least one computer hardwareprocessor to access second distance data determined by the distancesensing device at a second time occurring after the first time, whereinthe distance sensing device determined the second distance data with theobject at a second position within the field-of-view of the distancesensing device, wherein the first position is different than the secondposition, and the object is only partially in the field-of-view at thesecond position. The non-transitory computer-readable storage mediumstores processor-executable instructions that, when executed by at leastone computer hardware processor, cause the at least one computerhardware processor to determine a distance measurement model for theobject based on the first distance data and the second distance data,wherein the distance measurement model is configured to model a physicalparameter of the object within the field-of-view of the distance sensingdevice over time. The non-transitory computer-readable storage mediumstores processor-executable instructions that, when executed by at leastone computer hardware processor, cause the at least one computerhardware processor to determining, based on the first distance data, thesecond distance data, and the distance measurement model, third distancedata indicative of an estimated distance to the object prior to theobject being entirely within the field-of-view of the distance sensingdevice. The non-transitory computer-readable storage medium storesprocessor-executable instructions that, when executed by at least onecomputer hardware processor, cause the at least one computer hardwareprocessor to determine, based on the third distance data, dataindicative of a height of the object.

There has thus been outlined, rather broadly, the features of thedisclosed subject matter in order that the detailed description thereofthat follows may be better understood, and in order that the presentcontribution to the art may be better appreciated. There are, of course,additional features of the disclosed subject matter that will bedescribed hereinafter and which will form the subject matter of theclaims appended hereto. It is to be understood that the phraseology andterminology employed herein are for the purpose of description andshould not be regarded as limiting.

BRIEF DESCRIPTION OF DRAWINGS

The accompanying drawings are not intended to be drawn to scale. In thedrawings, each identical or nearly identical component that isillustrated in various figures is represented by a like numeral. Forpurposes of clarity, not every component may be labeled in everydrawing. In the drawings:

FIG. 1 is a diagram showing an exemplary machine vision systemapplication with an object moving on a conveyor belt towards the fieldof view of an image sensing device, according to some embodiments of thetechnology described herein;

FIG. 2 is a diagram showing the object from FIG. 1 moved further alongso that the object is partially in the field of view of the imagesensing device, according to some embodiments of the technologydescribed herein;

FIG. 3 is a diagram showing the object from FIG. 1 having moved furtheralong so that the object is further in the field of view of the imagesensing device compared to FIG. 2 , according to some embodiments of thetechnology described herein;

FIG. 4A is a diagram showing an exemplary method for determining andupdating a distance measurement model using range measurements,according to some embodiments of the technology described herein;

FIG. 4B is a diagram showing an exemplary detailed method fordetermining and updating a distance measurement model using distancesensing measurements, according to some embodiments of the technologydescribed herein;

FIG. 5A is a diagram showing an exemplary method for determining andupdating a distance measurement model using distance measurements andparameter estimation, according to some embodiments of the technologydescribed herein; and

FIG. 5B is a diagram showing an exemplary detailed method fordetermining and updating a distance measurement model using distancemeasurements and parameter estimation, according to some embodiments ofthe technology described herein.

DETAILED DESCRIPTION

As described above, the techniques described herein can be used topredict object features for a variety of machine vision applications.For example, the techniques can be used for logistic machine visionapplications, which include conveyor-based applications, sortingapplications, and/or the like. Some logistics applications use a fixedfocus lens that has a deep depth of field and a small aperture. As aresult of the small aperture, the images may be not well illuminatedbecause only limited light can pass through the lens. Alternatively,auto-focus techniques can be used instead of fixed focuses lenses. Sinceauto-focus techniques can use a lens with a shallower depth of field,more light can be captured, which can improve the image quality.Fast-focusing lenses are available, such as liquid lenses that canchange focus in 1-3 ms. Distance sensing devices, such as time-of-flightsensors, can be used to detect the height of an object, which can beused to drive the auto-focus of the lens. Time-of-flight sensors canoperate at rates of 3-5 KHz, for example.

While auto-focus techniques can improve image quality, the inventorshave discovered and appreciated that such techniques may not be able tofocus fast enough to allow the imaging device to capture a sufficientnumber of high-quality images of the object. For example, high speedlogistics applications can use conveyor belts that move objects at veryhigh rates, such as 600-800 feet per minute (or 10-13.5 feet persecond). Therefore, objects may only within the field of view of animaging device for less than a second, and maybe for less than 100-500ms. Imaging sensors typically operate at 50-60 Hz, which is around 15-20ms for each image. For high speed logistics applications, the imagingdevice may only be able to capture a few useful images of an objectwhile it is within the imaging sensor's field of view (e.g., such asonly 1-2 images of the full barcode on an object). This problem can befurther compounded for high objects that are closer to the imagingsensor, which can further limit opportunities to capture sufficientimages of the object (e.g., since the field of view is more limited forhigher objects, so the objects are in the field of view for less timethan shorter objects). The system can be further limited by the distancesensing device. For example, while time-of-flight sensors can operate athigh rates (e.g., such that the time-of-flight sensor can obtain 10-20time-of-flight readings between each image capture of a 50-60 Hz imagingdevice), time-of-flight sensors can only provide accurate heightmeasurements when the entire object is within the field-of-view of thetime-of-flight sensor. Therefore, for many machine vision systems, suchas systems that dispose the time-of-flight sensor at or near the imagesensor, time-of-flight sensors typically do not provide for fast-enoughheight measurement to drive auto-focus applications.

The inventors have therefore recognized and appreciated that it isdesirable to provide for fast object height detection to address theseand other problems with existing machine vision systems. For example,fast object height detection can be used to focus the lens assemblyprior to and/or just as the object (or a relevant portion of the object)moves into the field of view of the imaging device so that the imagingdevice has a maximum amount of time to capture images of the objectwhile it is within the imaging device's field of view. The inventorshave developed techniques to predict the height of a moving objectbefore the entire object is within the field of view of a distancesensing device. Such techniques can allow a machine vision system toinclude both the imaging device and the distance sensing device within asingle package and still allow the machine vision system to auto-focusthe lens assembly sooner than otherwise possible with previous distancesensing devices. For example, if a distance measuring device takes Nacquisitions of an object as it enters the field of view of the distancemeasuring device until the object entirely enters the field of view, thetechniques can estimate the object parameters 50% (e.g., ½×N feweracquisitions) sooner. Such techniques can additionally or alternativelyprovide greater accuracy than can otherwise be achieved with imperfectdata. For example, the techniques can perform auto-focus with just a fewnoisy distance measurements (e.g., only a few noisy distancemeasurements made before the object is completely within thefield-of-view of the system), which can provide for performingauto-focus at much faster conveyer speeds than previously possible.

In the following description, numerous specific details are set forthregarding the systems and methods of the disclosed subject matter andthe environment in which such systems and methods may operate, etc., inorder to provide a thorough understanding of the disclosed subjectmatter. In addition, it will be understood that the examples providedbelow are exemplary, and that it is contemplated that there are othersystems and methods that are within the scope of the disclosed subjectmatter.

FIG. 1 is a diagram 100 showing an exemplary machine vision systemapplication with an object 102 moving on a conveyor belt 104 towards thefield of view S 106 of an image sensing device 108, according to someembodiments of the technology described herein. In this example, theobject 102 has a height H 110, and the object 102 is moving on theconveyor belt 104 in direction 112. The image sensing device 108 is adistance D 114 from the conveyor belt 104. FIG. 2 is a diagram showingthe object 102 from FIG. 1 moved further along direction 112 so thatobject 102 is partially in the field of view S 106 of the image sensingdevice 108, according to some embodiments of the technology describedherein. As shown in FIG. 2 , a portion Sk1 200 of the top surface of theobject 102 is within the field of view S 106. FIG. 3 is a diagramshowing the object 102 from FIG. 1 having moved further along direction112 so that object 102 is further in the field of view S 106 of theimage sensing device 108 compared to FIG. 2 , according to someembodiments of the technology described herein. As shown in FIG. 3 , aportion Sk2 300 of the top surface of the object 102 is within the fieldof view S 106. The portion Sk2 300 is greater than the portion Sk1 200of FIG. 2 .

While FIGS. 1-3 just show one object 102, it should be appreciated thatmany objects can travel on the conveyor belt, and those objects may notbe of uniform shape, spacing, and/or the like. Therefore, in many visionsystem applications, such as symbology-decoding in logistics operations(e.g. tracking barcodes on packages as they pass down a conveyor line),the number, height, length, overall size and spacing gap between objectscan be highly variable.

In some embodiments, the distance sensing device is a time-of-flightsensor, such as an integrated single-point or multi-point time-of-flightsensor that predicts distance information. The time-of-flight sensor caninclude an emitter configured to emit a beam, such as a laser beam or alight pulse (e.g., an IR-based light pulse), and a receiver configuredto receive the reflected beam. The time-of-flight sensor can modulatethe intensity of the beam at a high frequency, such that there is aphase shift between the emitted beam and the reflected beam. Thetime-of-flight sensor can include circuitry to measure the degree ofphase shift by comparing the phase at the emitter with that at thereceiver. The measured phase shift is then converted to a distancemeasurement based on calibration parameters that reside within thesensor and/or external electronics.

While not shown, the distance sensing device 108 can be disposed nearthe imaging device, such as incorporated into a single package with theimaging device and/or mounted near the imaging device (e.g., mounted toa lens assembly of the imaging device). An exemplary configuration caninclude an imaging device integrated with a ToF sensor and/or anillumination device, such as the High-Powered Integrated Torch (HPIT)provided by Cognex Corporation, the assignee of the present application.Another exemplary configuration can include an imaging device and a ToFsensor mounted at a different location than the imaging device. In suchexamples, the imaging device and ToF sensor can be mounted with one ormore configuration constraints, such as with optical axes that areperpendicular to the conveyor, at a same height, with the ToF sensorupstream of the conveyor belt from the imaging device such that the ToFsensor can perform measurements before the object reaches the imagingdevice, and/or other system configuration constraints. In someembodiments, the imaging device can include one or more internalprocessors (e.g., FPGAs, DSPs, ARM processors, and/or the like) andother components that allow it to act as a standalone unit, providing adesired output data (e.g. decoded symbol information) to a downstreamprocess, such as an inventory tracking computer system or logisticsapplication. In some embodiments, the machine vision system can includeexternal processing functionality, such as a personal computer, laptop,and/or the like, that is configured to provide the machine vision systemprocessing functionality.

In some embodiments, the machine vision system can be an image-basedsymbology reader. The reader can include an imaging device with an imagesensor and optics and a vision system processor, which is arranged tofind and decode symbols (e.g., barcodes) in images acquired by the imagesensor. The distance sensing device 108 can be integrated with theimaging device, which predicts distance information as described herein(e.g., in conjunction with FIGS. 4A-5B) for the object 102 in the fieldof view of the image sensor. The time-of-flight sensor can beoperatively connected with at least one of the vision system processorand/or a imager control.

In some embodiments, the imaging device can include and/or be in opticalcommunication with a lens assembly. The choice of lens configuration canbe driven by a variety of factors, such as lighting/illumination, fieldof view, focal distance, relative angle of the camera axis and imagedsurface, and/or the size of the details on the imaged surface. In someexamples, the cost of the lens and/or the available space for mountingthe vision system can also drive the choice of lens. An exemplary lensconfiguration that can be desirable in certain vision systemapplications is an automatic focusing (auto-focus) assembly. By way ofexample, an auto-focus lens can be facilitated by a so-called liquidlens assembly.

The use of a liquid lens can simplify installation, setup andmaintenance of the vision system by eliminating the need to manuallytouch or adjust the lens. Relative to other auto-focus mechanisms, theliquid lens can have fast response times. Liquid lenses can also be usedfor applications with reading distances that change fromobject-to-object (surface-to-surface) or during the changeover from thereading of one object to another object, such as scanning a movingconveyor containing differing sized/height objects (e.g., such asshipping boxes). While the example shown in FIGS. 1-3 shows a conveyorbelt machine vision application, it should be appreciated that it can bedesirable to quickly focus for imaging in many different vision systemapplications, and the techniques described herein are therefore notlimited to such exemplary embodiments.

Liquid lenses can be implemented in different manners. One exemplaryliquid lens embodiment uses water and oil, and can essentially shape theoil into a lens using an applied voltage. The variation of voltagepassed through the lens by surrounding circuitry leads to a change ofcurvature of the oil-water interface, which in turn leads to a change ofthe focal length of the lens. Another exemplary liquid lens embodimentuses a movable membrane covering a liquid reservoir to vary its focaldistance. A bobbin can exerts pressure to alter the shape of themembrane and thereby vary the lens focus. The bobbin can be moved byvarying the input current within a preset range. Differing currentlevels can provide differing focal distances for the liquid lens.

As described herein, the range/distance information from the distancesensing device can be processed to auto-focus a variable (e.g. liquid)lens during runtime operation based on the particular size/shapeobject(s) within the field of view and before the object(s) are fullywithin the field of view of the time-of-flight sensor. For example,predicted distance information can be used to set the focus distance ofthe lens of the imaging device prior to the object being partiallyand/or entirely within the field of view of the imaging device. In someembodiments, the system is configured such that the distance measuringdevice has a higher speed than the than the imaging sensor. For example,a 50 Hz imaging sensor can be used with a 4 KHz time-of-flight sensor.

FIG. 4A is a diagram showing an exemplary method 400 for determining andupdating a distance measurement model using range measurements,according to some embodiments of the technology described herein. Themachine vision system can perform the method 400 to predict features ofobjects (e.g., height, area, etc.) prior to the objects being entirelywithin the field of view of a distance sensing device. For example,referring to FIGS. 2 and 3 , the method 400 can be used to predict theheight H 110 of the object 102 prior to the object 102 being entirelywithin the field of view S of the distance sensing device 108. At step402, the machine vision system determines a feature measurement model,such as a distance measurement model that models the distancemeasurements of the distance sensing device to the object over time asthe object enters the field of view of the distance sensing device topredict the height of the object prior to the object being entirelywithin the field of view. As described herein, the machine vision systemcan determine the distance measurement model based on predeterminedparameters and/or based on initial distance measurements of the objectas the object begins to enter the field of view.

The machine vision system continues to acquire distance data over time,and to update the distance measurement model accordingly. At step 404,the machine vision system acquires and/or processes new distancemeasurement data. At step 406, the machine vision system updates thedistance measurement model state 406 based on the distance measurementdata acquired at step 404. The method proceeds back to step 404 andprocesses and/or acquires new distance measurement data. The methodperforms steps 404 and 406 until the distance measurement modelsufficiently converges with the distance measurement data. For example,as described further herein, the machine vision system can determinethat the distance measurement model converges with the time-of-flightmeasurement data when changes in the distance measurement model and/orthe estimated object height determined at step 406 is below apredetermined threshold.

FIG. 4B is a diagram showing an exemplary detailed method 450 fordetermining and updating a distance measurement model using distancesensing measurements, according to some embodiments of the technologydescribed herein. At step 452, the machine vision system accesses firstdistance data determined by a distance sensing device (e.g., atime-of-flight sensor) at a first time. The distance sensing devicecaptured and/or determined the first distance data with an object at afirst position that is only partially within the field-of-view of thedistance sensing device. For example, referring to FIG. 2 , the distancesensing device 108 captured the first distance data with the object 102only partially within the field-of-view S 106, as shown by Sk1 200.

At step 454, the machine vision system accesses second distance datadetermined by the distance sensing device at a second, subsequent timeafter the first time. The distance sensing device captured and/ordetermined the second distance data with the object at a second positionthat is still only partially within the field-of-view of the distancesensing device. Since the distance data is determined over time as theobject moves, the first position associated with the first distance datais different than the second position associated with the seconddistance data. For example, referring to FIG. 3 , the distance sensingdevice 108 captured the second distance data with the object 102 onlypartially within the field-of-view S 106, as shown by Sk2 300. Comparedto FIG. 2 , while the object 102 is only partially within thefield-of-view S 106 in both FIGS. 2-3 , the object 102 is further withinthe field-of-view S 106 in FIG. 3 than in FIG. 2 (and hence Sk2 300 isgreater than Sk1 200).

As described herein, the distance sensing device can be a time-of-flightsensor that is part of a machine vision system that also includesinternal and/or external processing hardware and associated software toperform machine vision tasks. Referring to steps 452 and 454, accessingthe first and second distance data can include the processing hardware(e.g., which is part of the imaging device) receiving first and secondtime-of-flight measurements from the time-of-flight sensor. As anotherexample, the processing hardware can access the time-of-flightmeasurements from a memory shared with the time-of-flight sensor.

At step 456, the machine vision system determines a distance measurementmodel for the object based on the first distance data and the seconddistance data. The distance measurement model is configured to model aphysical parameter of the object within the field-of-view of thedistance sensing device over time, such as the height of the object, thesurface area of the object, and/or the like. At step 458, the machinevision system determines, based on the previous distance data (e.g., thefirst distance data and the second distance data for the firstexecution) and the distance measurement model, distance data that isindicative of an estimated distance to the object prior to the objectbeing entirely within the field-of-view of the distance sensing device.

At step 460, the machine vision system determines whether the distancemeasurement model converges with the distance measurement data. In someembodiments, the machine vision system can determine that changes in thedistance measurement model are below a predetermined threshold. In someembodiments, the machine vision system can determine that changesbetween the distance measurement data and the estimated object height(e.g., the data determined at step 458) is below a predeterminedthreshold. For example, as described further in conjunction withEquations 1-2, the object height can be modeled as a parameter that ispart of the observation matrix and the observation matrix can be updatedeach iteration. If, after a number of iterations, the predicted heightis close to the observation (e.g., within a threshold), the system candetermine that the model used at the current iteration is sufficient andcan use the observation matrix to determine the height of the object(e.g., such that the distance measurement model is stable at thisstate). If the machine vision system determines that the distancemeasurement model does not converge with the distance measurement data,the method 450 proceeds to step 462 and updates the distance measurementmodel with one or more additional distance measurements, and proceedsback to step 458.

If the machine vision system determines that the distance measurementmodel converges with the distance measurement data, the method 450proceeds to step 464 and the machine vision system determines, based onthe distance data determined at step 458, data indicative of a height ofthe object. For example, the machine vision system can use thelast-determined data from step 458 as being indicative of the objectheight.

As described herein, the machine vision system can use the dataindicative of the height of the object to perform various tasks of themachine vision system. For example, the machine vison system candetermine, based on the distance data, data indicative of a focusadjustment for the lens of the imaging device associated with thedistance sensing device. The machine vision system can transmit, basedon the data indicative of the focus adjustment, one or more signals tochange a focus of the lens. In some embodiments, the machine visionsystem uses the focus adjustment to change the focus of a liquid lens.The machine vision system can capture an image of the object aftertransmitting the one or more signals to change the focus of the lens. Insome embodiments, the machine vision system can be configured to wait apredetermined amount of time after transmitting the one or more signalsto change the focus of the lens prior to capturing the image of theobject. In some embodiments, the system can receive feedback data fromthe liquid lens assembly indicative of the focus adjustment beingcompleted, and the machine vision system can capture the imageresponsive to receipt of the feedback data.

In some embodiments, the machine vision system can determine, based onthe estimated distance data, data indicative of a brightness adjustmentfor an illumination module of the imaging device. For example, if thedistance data is indicative of a short object, the machine vision systemmay be configured to user a higher brightness setting for theillumination module compared to when the distance data is indicative ofa higher object. Therefore, the machine vision system can be configuredto use brighter illumination for objects further away from the imagingdevice, and softer illumination for objects that are closer to theimaging device. In some embodiments, to adjust the illumination ofobjects the techniques can adjust the exposure time of the imagingdevice to adjust image brightness (e.g., without adjusting theillumination module). For example, the system can reduce exposure timefor objects closer to the camera.

In some embodiments, the machine vision system can access dataindicative of at least one constraint the machine vision application.For example, the machine vision system can access data indicative of amotion parameter of the object that is associated with the motion of thebox through the field-of-view of the time-of-flight sensor. The machinevision system can use the data indicative of the motion parameter(s) todetermine the distance data. For example, referring to FIG. 4B, at step458 the machine vision system can determine the distance data based onthe first distance data, the second distance data, the distancemeasurement model, and the one or more motion parameters. In someembodiments, the motion parameter includes a velocity of the objectand/or an acceleration of the object as the object travels through thefield-of-view of the distance sensing device. For example, the machinevision system can determine the distance data based on the firstdistance data, the second distance data, the distance measurement model,and the velocity of the object.

In some embodiments, the machine vision system can use a Kalman filterto model the distance to the object as the object enters the field ofview of the distance sensing device. For example, the machine visionsystem can use a Kalman filter to estimate the object of the area overtime. Referring to step 458 in FIG. 4B, for example, the machine visionsystem can determine, using the distance measurement model, estimates ofthe object area based on the acquired distance data (e.g., acquired atsteps 452, 454 and/or 462). For example, the machine vision system candetermine a first object area estimate of the object (which is onlypartially within the field-of-view of the distance sensing device) basedon the first distance data. The machine vision system can use the Kalmanfilter to determine subsequent object area estimates. For example, themachine vision system can determine, using the distance measurementmodel, a second object area estimate of the object (which is still onlywithin the field-of-view of the distance sensing device) based on thefirst object area estimate and the second distance data. The machinevision system can determine, based on the second object area estimate, aheight estimate of the object.

The machine vision system can be configured to use one or more equationsto measure the state (e.g., the area of the object within the field ofview) and/or perform a measurement update (e.g., to update the estimateddistance to the object) of the distance sensing device. ExemplaryEquation 1 below can be used to perform a state update:S _(k) =S _(k-1) +S _(Δ) +w _(k)  Equation 1

Where:

S_(k) is the area of the top surface of the object within the field ofview of the distance sensing device at the current time k;

S_(k-1) is the area of the top surface of the object within the field ofview of the distance sensing device at the previous time k−1;

S_(Δ) is the difference between S_(k) and S_(k-1); and

w_(k) is a model for system noise, random variation, and/or measurementnoise/inaccuracy.

As shown in Equation 1, the states can be the object area in thefield-of-view of the distance determining device. The states can bemodeled based on parameters of the machine vision system. For example,if the objects move at a near-constant speed (such as on a conveyorbelt) and the frame rate of the TOF sensor is sufficiently high enough,the state update can be modeled with a linear function in which thechange of object area is constant in unit time. In some embodiments, thefirst two object area measurements can be initial values determinedbased on the machine vision application specification, such as box sizerange, conveyor belt width, object moving speed, and/or the like. Itshould be appreciated the initial object area measurements need not beaccurate. For example, w_(k) can be used to model noise from thephysical world, such as the non-consistent speed of the conveyor belt,etc. The values for w_(k) (e.g., and v_(k), below) can be modeled interms of the system noise and an observation covariance matrix. Theparameter w_(k) can be assigned an initial value based on theapplication (e.g., based on expected variations of the conveyor speed,the accuracy of the TOF reading, etc.). The initial value(s) need notnecessarily be accurate, but can be updated for each iteration toimprove accuracy using the Kalman filter. From the third state (andonwards), the object areas and/or height are obtained and determinedfrom the model using real-time distance measurements, and the model canbe update for each new distance measurement.

Equation 2 provides an exemplary equation for performing a distancemeasurement:TOF_(k)=−(h/S)S _(k) +d+v _(k)  Equation 2

Where:

TOF_(k) is the time-of-flight measurement at time k;

h is the height of the object;

S is the field-of-view of the time-of-flight sensor;

S_(k) is the area of the top surface of the object within the field ofview of the distance sensing device at the current time k;

d is the distance between the time-of-flight sensor and the conveyorbelt; and

v_(k) is a model for observation noise in the distance measurement(e.g., noise that comes from the inaccuracy of the TOF reading).

Referring to Equation 2, the TOF sensor reading can be determined basedon a weighted average of the object height h and the distance d.

It should be appreciated that Equations 1 and 2 are provided forillustrative purposes only, and that other models can be used for theparticular machine vision application. For example, other formats of theKalman filter model can be used depending on whether the object movingis subject to acceleration(s), whether the object area changes, and/orother variables. Over time, measurements of the time-of-flight sensorcan slowly decrease (e.g., from measuring a distance to the conveyorbelt to measuring a distance to the top of the object) as the objectcomes into the field-of-view of the time-of-flight sensor. The estimatedbox area that can be determined by the model can correspondingly slowlyincrease over time to be inversely proportional to the time-of-flightmeasurements, such that the estimated box area increases over time.

In some embodiments, the machine vision system can use parameterestimation techniques (e.g., Expectation Maximization (EM), Maximum APosteriori (MAP), and/or the like) to update parameters of the model.For example, as shown in exemplary Equation 2, the object height can bepart of the observation matrix. A parameter estimation algorithm can beused at each iteration to update the box height. FIG. 5A is a diagramshowing an exemplary method 500 for determining and updating a distancemeasurement model using distance measurements and parameter estimation,according to some embodiments of the technology described herein. Atstep 502, the machine vision system determines a distance measurementmodel using the techniques described herein. Each time the systemobtains a new distance observation at step 504, at step 506 the systemuses the model (e.g., a Kalman filter) to update the state of thedistance measurement model, and at step 508 the machine vision systemuses parameter estimation to update the system observation matrix.

FIG. 5B is a diagram showing an exemplary detailed method 550 fordetermining and updating a distance measurement model using distancemeasurements and parameter estimation, according to some embodiments ofthe technology described herein. At step 552, the machine vision systemaccesses first distance data determined at a first time with an objectat a first position that is only partially within the field-of-view ofthe distance sensing device (e.g., as shown in FIG. 2 ). At step 554,the machine vision system accesses second distance data determined at asecond, subsequent time with the object at a second position that isstill only partially within the field-of-view of the distance sensingdevice (e.g., as shown in FIG. 3 ). At step 556, the machine visionsystem determines a distance measurement model for the object based onthe first distance data and the second distance data.

At step 558, the machine vision system determines, based on the previousdistance data (e.g., the first distance data and the second distancedata for the first execution) and the distance measurement model,distance data that is indicative of an estimated distance to the objectprior to the object being entirely within the field-of-view of thedistance sensing device. At step 560, the machine vision systemdetermines whether the distance measurement model converges with thedistance measurement data. If the machine vision system determines thatthe distance measurement model does not converge with the distancemeasurement data, the method 550 proceeds to step 562 and updates theparameters using parameter estimation (e.g., using EM, MAP, and/oranother parameter estimation technique). The method proceeds to step 564and updates the distance measurement model with one or more additionaldistance measurements, and proceeds back to step 558.

If the machine vision system determines that the distance measurementmodel converges with the distance measurement data, the method 550proceeds from step 560 to step 564 and the machine vision systemdetermines, based on the distance data determined at step 558, dataindicative of a height of the object. For example, the machine visionsystem can use the last-determined data from step 558 as beingindicative of the object height.

As described herein, measurements of the time-of-flight sensor canslowly decrease (e.g., from measuring a distance to the conveyor belt tomeasuring a distance to the top of the object) as the object comes intothe field-of-view of the time-of-flight sensor. The estimated box areadetermined using the distance measurement model can correspondinglyslowly increase over time to be inversely proportional to thetime-of-flight measurements, such that the estimated box area increasesover time. In some embodiments, using parameter estimation can fine-tunethe relation between the time-of-flight measurements and the box areameasurements.

The predicted distance information generated by the distance sensingdevice can also be employed to perform other aspects of the machinevision system. In some embodiments, the machine vision system can usethe predicted distance information to self-trigger image acquisition ofan object by the vision system. For example, when the system determinesa change in object distance (for example, a change in height from thesupporting base/conveyor belt), the system can trigger an image capture.In some embodiments, the machine vision system can use the predicteddistance information to perform object (e.g. box) size dimensioning.During calibration, the machine vision system can measure the distance D114 between the conveyor belt and vision system and store theinformation. At runtime, the machine vision system can capture imagesand can predict the distance to the object prior to the object beingentirely within the image and/or measure the distance to the object inthe center of the image/field of view. If the system detects arectangular shape, the dimensions of that rectangle can be determinedbased on the predicted and/or measured distance (e.g., and based onknown optical properties of imager, such as the image sensor size, lensfocal length, etc.). As shown in FIGS. 1-3 , the height 110 of object102 can be representative of essentially the difference between theheight of the imaging device from conveyor belt 104 and a predictedand/or measured shortest distance between the imaging device and theobject 102. In some embodiments, since the imaging device is fixedlymounted to the time-of-flight sensor, the height between the imagingdevice and object 102 can be represented by distance D 114.

In some embodiments, the machine vision system can be used to detect andanalyze object defects. For example, after the machine vision systemdetects the (e.g., rectangular) top surface as described herein, themachine vision system can measure deviations from the top surface shapeto detect damaged objects (e.g., damaged boxes). In some embodiments,the system can perform region of interest (RoI) detection. For example,the field of view of a camera-based symbology reader can be imaged ontothe sensed area of a multi-point (e.g., n×1 or n×m) time-of-flightsensor array. The sensor array can measure a 3D height map to narrow theregion of interest for symbology decoding. Determining in which part ofthe image the object resides can reduce decoding time because symbolcandidate features can be searched from a narrowed region of interest inthe overall acquired image.

In some embodiments, the machine vision system can detect a gap betweenobjects in the field of view, which can assist in linking symbol codesto the appropriate imaged object. For example, in logistics applicationswhere there can be more than one object entering and/or within the fieldof view at a time, the time-of-flight measurements can assist inlocating the edge(s) of the object and to determine what symbol isassociated with each object in the field of view.

In some embodiments, the machine vision system can use the distancepredicted and/or measured by the time-of-flight sensor to limit thereading range of the vision system to prevent unintentional reading. Forexample, if the distance to the object is within a defined range, thenthe machine vision system (a) captures an image, (b) initiates a symboldecoding process, and/or (c) transmits the decoded data for furtherprocessing.

Techniques operating according to the principles described herein may beimplemented in any suitable manner. For example, the embodiments may beimplemented using hardware, software or a combination thereof. Whenimplemented in software, the software code can be executed on anysuitable processor or collection of processors, whether provided in asingle computer or distributed among multiple computers. Such processorsmay be implemented as integrated circuits, with one or more processorsin an integrated circuit component, including commercially availableintegrated circuit components known in the art by names such as CPUchips, GPU chips, FPGA chips, microprocessor, microcontroller, orco-processor. Alternatively, a processor may be implemented in customcircuitry, such as an ASIC, or semicustom circuitry resulting fromconfiguring a programmable logic device. As yet a further alternative, aprocessor may be a portion of a larger circuit or semiconductor device,whether commercially available, semi-custom or custom. As a specificexample, some commercially available microprocessors have multiple coressuch that one or a subset of those cores may constitute a processor.Though, a processor may be implemented using circuitry in any suitableformat.

Further, it should be appreciated that a computer may be embodied in anyof a number of forms, such as a rack-mounted computer, a desktopcomputer, a laptop computer, or a tablet computer. Additionally, acomputer may be embedded in a device not generally regarded as acomputer but with suitable processing capabilities, including a PersonalDigital Assistant (PDA), a smart phone or any other suitable portable orfixed electronic device.

Also, a computer may have one or more input and output devices. Thesedevices can be used, among other things, to present a user interface.Examples of output devices that can be used to provide a user interfaceinclude printers or display screens for visual presentation of outputand speakers or other sound generating devices for audible presentationof output. Examples of input devices that can be used for a userinterface include keyboards, and pointing devices, such as mice, touchpads, and digitizing tablets. As another example, a computer may receiveinput information through speech recognition or in other audible format.In the embodiment illustrated, the input/output devices are illustratedas physically separate from the computing device. In some embodiments,however, the input and/or output devices may be physically integratedinto the same unit as the processor or other elements of the computingdevice. For example, a keyboard might be implemented as a soft keyboardon a touch screen. Alternatively, the input/output devices may beentirely disconnected from the computing device, and functionallyintegrated through a wireless connection.

Such computers may be interconnected by one or more networks in anysuitable form, including as a local area network or a wide area network,such as an enterprise network or the Internet. Such networks may bebased on any suitable technology and may operate according to anysuitable protocol and may include wireless networks, wired networks orfiber optic networks.

Also, the various methods or processes outlined herein may be coded assoftware that is executable on one or more processors that employ anyone of a variety of operating systems or platforms. Additionally, suchsoftware may be written using any of a number of suitable programminglanguages and/or programming or scripting tools, and also may becompiled as executable machine language code or intermediate code thatis executed on a framework or virtual machine.

In this respect, the invention may be embodied as a computer readablestorage medium (or multiple computer readable media) (e.g., a computermemory, one or more floppy discs, compact discs (CD), optical discs,digital video disks (DVD), magnetic tapes, flash memories, circuitconfigurations in Field Programmable Gate Arrays or other semiconductordevices, or other tangible computer storage medium) encoded with one ormore programs that, when executed on one or more computers or otherprocessors, perform methods that implement the various embodiments ofthe invention discussed above. As is apparent from the foregoingexamples, a computer readable storage medium may retain information fora sufficient time to provide computer-executable instructions in anon-transitory form. Such a computer readable storage medium or mediacan be transportable, such that the program or programs stored thereoncan be loaded onto one or more different computers or other processorsto implement various aspects of the present application as discussedabove. As used herein, the term “computer-readable storage medium”encompasses only a computer-readable medium that can be considered to bea manufacture (i.e., article of manufacture) or a machine. Alternativelyor additionally, the invention may be embodied as a computer readablemedium other than a computer-readable storage medium, such as apropagating signal.

The terms “code”, “program” or “software” are used herein in a genericsense to refer to any type of computer code or set ofcomputer-executable instructions that can be employed to program acomputer or other processor to implement various aspects of the presentapplication as discussed above. Additionally, it should be appreciatedthat according to one aspect of this embodiment, one or more computerprograms that when executed perform methods of the present applicationneed not reside on a single computer or processor, but may bedistributed in a modular fashion amongst a number of different computersor processors to implement various aspects of the present application.

Computer-executable instructions may be in many forms, such as programmodules, executed by one or more computers or other devices. Generally,program modules include routines, programs, objects, components, datastructures, etc. that perform particular tasks or implement particularabstract data types. Typically the functionality of the program modulesmay be combined or distributed as desired in various embodiments.

Also, data structures may be stored in computer-readable media in anysuitable form. For simplicity of illustration, data structures may beshown to have fields that are related through location in the datastructure. Such relationships may likewise be achieved by assigningstorage for the fields with locations in a computer-readable medium thatconveys relationship between the fields. However, any suitable mechanismmay be used to establish a relationship between information in fields ofa data structure, including through the use of pointers, tags or othermechanisms that establish relationship between data elements.

Various aspects of the present application may be used alone, incombination, or in a variety of arrangements not specifically discussedin the embodiments described in the foregoing and is therefore notlimited in its application to the details and arrangement of componentsset forth in the foregoing description or illustrated in the drawings.For example, aspects described in one embodiment may be combined in anymanner with aspects described in other embodiments.

Also, the invention may be embodied as a method, of which an example hasbeen provided. The acts performed as part of the method may be orderedin any suitable way. Accordingly, embodiments may be constructed inwhich acts are performed in an order different than illustrated, whichmay include performing some acts simultaneously, even though shown assequential acts in illustrative embodiments.

The indefinite articles “a” and “an,” as used herein in thespecification and in the claims, unless clearly indicated to thecontrary, should be understood to mean “at least one.”

The phrase “and/or,” as used herein in the specification and in theclaims, should be understood to mean “either or both” of the elements soconjoined, i.e., elements that are conjunctively present in some casesand disjunctively present in other cases. Multiple elements listed with“and/or” should be construed in the same fashion, i.e., “one or more” ofthe elements so conjoined. Other elements may optionally be presentother than the elements specifically identified by the “and/or” clause,whether related or unrelated to those elements specifically identified.Thus, as a non-limiting example, a reference to “A and/or B”, when usedin conjunction with open-ended language such as “comprising” can refer,in one embodiment, to A only (optionally including elements other thanB); in another embodiment, to B only (optionally including elementsother than A); in yet another embodiment, to both A and B (optionallyincluding other elements); etc.

As used herein in the specification and in the claims, the phrase “atleast one,” in reference to a list of one or more elements, should beunderstood to mean at least one element selected from any one or more ofthe elements in the list of elements, but not necessarily including atleast one of each and every element specifically listed within the listof elements and not excluding any combinations of elements in the listof elements. This definition also allows that elements may optionally bepresent other than the elements specifically identified within the listof elements to which the phrase “at least one” refers, whether relatedor unrelated to those elements specifically identified. Thus, as anon-limiting example, “at least one of A and B” (or, equivalently, “atleast one of A or B,” or, equivalently “at least one of A and/or B”) canrefer, in one embodiment, to at least one, optionally including morethan one, A, with no B present (and optionally including elements otherthan B); in another embodiment, to at least one, optionally includingmore than one, B, with no A present (and optionally including elementsother than A); in yet another embodiment, to at least one, optionallyincluding more than one, A, and at least one, optionally including morethan one, B (and optionally including other elements); etc.

Use of ordinal terms such as “first,” “second,” “third,” etc., in theclaims to modify a claim element does not by itself connote anypriority, precedence, or order of one claim element over another or thetemporal order in which acts of a method are performed, but are usedmerely as labels to distinguish one claim element having a certain namefrom another element having a same name (but for use of the ordinalterm) to distinguish the claim elements.

Also, the phraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting. The use of“including,” “comprising,” or “having,” “containing,” “involving,” andvariations thereof herein, is meant to encompass the items listedthereafter and equivalents thereof as well as additional items.

What is claimed is:
 1. A computerized method, comprising: accessingfirst distance data determined by a distance sensing device at a firsttime, wherein the distance sensing device determined the first distancedata with an object at a first position within a field-of-view of thedistance sensing device, wherein the object is only partially in thefield-of-view at the first position; accessing second distance datadetermined by the distance sensing device at a second time occurringafter the first time, wherein the distance sensing device determined thesecond distance data with the object at a second position within thefield-of-view of the distance sensing device, wherein the first positionis different than the second position, and the object is only partiallyin the field-of-view at the second position; determining a distancemeasurement model for the object based on the first distance data andthe second distance data, wherein the distance measurement model isconfigured to model a physical parameter of the object to predict thephysical parameter of the object when the object moves within thefield-of-view of the distance sensing device over time, including futuretimes that will occur after the first and second times; determining,based on the first distance data, the second distance data, and thedistance measurement model, third distance data indicative of anestimated distance to the object, wherein the estimated distancepredicts a distance to the object at a future time when the object willbe within the field-of-view of the distance sensing device prior to theobject actually being entirely within the field-of-view of the distancesensing device; and determining, based on the third distance data, dataindicative of a height of the object.
 2. The computerized method ofclaim 1, wherein the physical parameter of the object comprises an areaof the object, a height of the object, or some combination thereof. 3.The computerized method of claim 1, further comprising: determining,based on the third distance data, data indicative of a focus adjustmentfor a lens of an imaging device associated with the distance sensingdevice.
 4. The computerized method of claim 3, further comprisingtransmitting, based on the data indicative of the focus adjustment, oneor more signals to change a focus of the lens.
 5. The computerizedmethod of claim 4, further comprising capturing an image of the objectafter transmitting the one or more signals to change the focus of thelens.
 6. The computerized method of claim 1, further comprising:determining, based on the third distance data, data indicative of abrightness adjustment for an illumination module of an imaging deviceassociated with the distance sensing device.
 7. The computerized methodof claim 1, further comprising: accessing data indicative of a motionparameter of the object; and determining the third distance datacomprises determining the third distance data based on the firstdistance data, the second distance data, the distance measurement model,and the motion parameter.
 8. The computerized method of claim 7, whereinthe motion parameter comprises a velocity of the object as the objecttravels through the field-of-view of the distance sensing device.
 9. Thecomputerized method of claim 8, wherein the motion parameter comprisesan acceleration of the object as the object travels through thefield-of-view of the distance sensing device.
 10. The computerizedmethod of claim 8, wherein determining the third distance data comprisesdetermining the third distance data based on the first distance data,the second distance data, the distance measurement model, and thevelocity of the object.
 11. The computerized method of claim 1, whereinthe distance sensing device comprises a time-of-flight sensor, andaccessing the first and second distance data comprises receiving firstand second time-of-flight measurements from the time-of-flight sensor.12. The computerized method of claim 1, wherein determining the thirddistance data based on the first distance data, the second distancedata, and the distance measurement model comprises: determining, usingthe distance measurement model, a first object area estimate of a firstarea of the object within the field-of-view of the distance sensingdevice based on the first distance data; and determining, using thedistance measurement model, a second object area estimate of a secondarea of the object within the field-of-view of the distance sensingdevice based on the first object area estimate and the second distancedata.
 13. The computerized method of claim 12, wherein the distancemeasurement model is a Kalman filter.
 14. The computerized method ofclaim 12, further comprising determining, based on the second objectarea estimate, a height estimate of the object.
 15. The computerizedmethod of claim 14, wherein determining the height estimate of theobject comprises determining, using a parameter estimation algorithm,the height estimate based on the second object area estimate.
 16. Thecomputerized method of claim 15, wherein the parameter estimationalgorithm is an Expectation Maximization (EM) algorithm, a Maximum APosteriori (MAP) algorithm, or some combination thereof.
 17. Thecomputerized method of claim 1, further comprising: updating, based onthe third distance data, a focus adjustment for a lens of an imagingdevice associated with the distance sensing device; and capturing animage of the object after updating the focus adjustment for the lens.18. The computerized method of claim 1, further comprising: adjusting,based on the third distance data, a brightness setting for anillumination module of an imaging device associated with the distancesensing device; and capturing an image of the object after adjusting thebrightness setting for the illumination module.
 19. At least onenon-transitory computer-readable storage medium storingprocessor-executable instructions that, when executed by at least onecomputer hardware processor, cause the at least one computer hardwareprocessor to perform the acts of: accessing first distance datadetermined by a distance sensing device at a first time, wherein thedistance sensing device determined the first distance data with anobject at a first position within a field-of-view of the distancesensing device, wherein the object is only partially in thefield-of-view at the first position; accessing second distance datadetermined by the distance sensing device at a second time occurringafter the first time, wherein the distance sensing device determined thesecond distance data with the object at a second position within thefield-of-view of the distance sensing device, wherein the first positionis different than the second position, and the object is only partiallyin the field-of-view at the second position; determining a distancemeasurement model for the object based on the first distance data andthe second distance data, wherein the distance measurement model isconfigured to model a physical parameter of the object to predict thephysical parameter of the object when the object moves within thefield-of-view of the distance sensing device over time, including futuretimes that will occur after the first and second times; determining,based on the first distance data, the second distance data, and thedistance measurement model, third distance data indicative of anestimated distance to the object, wherein the estimated distancepredicts a distance to the object at a future time when the object willbe within the field-of-view of the distance sensing device prior to theobject actually being entirely within the field-of-view of the distancesensing device; and determining, based on the third distance data, dataindicative of a height of the object.
 20. The computer readable mediumof claim 19, wherein the processor-executable instructions further causethe at least one computer hardware processor to perform the acts of:accessing data indicative of a motion parameter of the object; anddetermining the third distance data comprises determining the thirddistance data based on the first distance data, the second distancedata, the distance measurement model, and the motion parameter.
 21. Anapparatus comprising a processor in communication with a memory, theprocessor being configured to execute instructions stored in the memorythat cause the processor to: access first distance data determined by adistance sensing device at a first time, wherein the distance sensingdevice determined the first distance data with an object at a firstposition within a field-of-view of the distance sensing device, whereinthe object is only partially in the field-of-view at the first position;access second distance data determined by the distance sensing device ata second time occurring after the first time, wherein the distancesensing device determined the second distance data with the object at asecond position within the field-of-view of the distance sensing device,wherein the first position is different than the second position, andthe object is only partially in the field-of-view at the secondposition; determine a distance measurement model for the object based onthe first distance data and the second distance data, wherein thedistance measurement model is configured to model a physical parameterof the object to predict the physical parameter of the object when theobject moves within the field-of-view of the distance sensing deviceover time, including future times that will occur after the first andsecond times; determine, based on the first distance data, the seconddistance data, and the distance measurement model, third distance dataindicative of an estimated distance to the object, wherein the estimateddistance predicts a distance to the object at a future time when theobject will be within the field-of-view of the distance sensing deviceprior to the object actually being entirely within the field-of-view ofthe distance sensing device; and determine, based on the third distancedata, data indicative of a height of the object.
 22. The apparatus ofclaim 21, wherein the processor is further configured to executeinstructions stored in the memory that cause the processor to: accessdata indicative of a motion parameter of the object; and determine thethird distance data comprises determining the third distance data basedon the first distance data, the second distance data, the distancemeasurement model, and the motion parameter.